The present disclosure relates to an image processing apparatus, an image processing method, and a program. In particular, the present disclosure relates to an image processing apparatus, an image processing method, and a program that generate images with a high dynamic range (wide dynamic range).
A solid-state image pickup device such as a CCD image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor used in a video camera or a digital still camera carries out photoelectric conversion by accumulating charge in keeping with the amount of incident light and outputting an electrical signal corresponding to the accumulated charge. However, there is a limit on the amount of charge that can be accumulated in a photoelectric conversion element, so that when a certain amount of light has been received, a saturation level is reached, resulting in regions of a subject with a certain brightness or higher being set at a saturation luminance level, a problem referred to as “blown out highlights” or “clipping”.
To prevent clipping, processing is carried out to control the charge accumulation period of the photoelectric conversion element in accordance with the change in light from outside or the like to adjust the exposure length and thereby optimize sensitivity. For example, by using a high shutter speed to shorten the exposure length for a bright subject, the charge accumulation period of the photoelectric conversion element is reduced and an electrical signal is outputted before the amount of accumulated charge reaches the saturation level. By carrying out such processing, it is possible to output an image in which tones are correctly reproduced for the subject.
However, if a high shutter speed is used when photographing a subject in which both bright and dark regions are present, the exposure length will not be sufficient for the dark regions, which will result in deterioration in the S/N ratio and a fall in image quality. To correctly reproduce the luminance levels of bright regions and dark regions in a photographed image of a subject that includes both bright and dark regions, it is necessary to use a long exposure for pixels on the image sensor where there is little incident light to achieve a high S/N ratio and to carry out processing to avoid saturation for pixels with large amounts of incident light.
As a method of realizing such processing, a method that consecutively picks up a plurality of images with different exposure lengths and blends the images is known. That is, there is a method that separately and consecutively picks up a long exposure image and a short exposure image and carries out a blending process using the long exposure image for dark image regions and the short exposure image for bright image regions with blown-out highlights in the long exposure image to generate a single image. In this way, by combining a plurality of images with different exposures, it is possible to obtain an image with the wide dynamic range with no blown-out highlights, that is, a high-dynamic range (HDR) image.
As one example, Japanese Laid-Open Patent Publication No. 2000-50151 discloses a configuration that picks up two images in which a plurality of different exposure lengths are set and obtains an HDR image by combining such images. Such processing will now be described with reference to FIG. 1. An image pickup device outputs image data with two different exposure lengths within a video rate (30-60 fps) during pickup of moving images, for example. During the pickup of still images also, image data with two different exposure lengths is generated and outputted. FIG. 1 is a diagram useful in explaining the characteristics of images with two different exposure lengths (that is, a long exposure image and a short exposure image) generated by the image pickup device. The horizontal axis shows time (t) and the vertical axis shows accumulated charge (e) in a light-receiving photodiode (PD) that constructs a photoelectric conversion element corresponding to one pixel of a solid-state image pickup element.
For example, if a large amount of light has been received by the light-receiving photodiode (PD), that is, when the subject is bright, there is a sudden rise in the accumulated charge over time as shown in a high luminance region 11 shown in FIG. 1. Meanwhile, if little light has been received by the light-receiving photodiode (PD), that is, when the subject is dark, there is a gradual rise in the accumulated charge over time as shown in a low luminance region 12 in FIG. 1.
The time t0 to t3 corresponds to the exposure length TL for acquiring the long exposure image. Even when this long exposure length TL is set, as can be understood from the line showing the low luminance region 12, the accumulated charge will not reach the saturation level even at time t3 (the non-saturated point Py), which means accurate tone reproduction is achieved from a tone level of a pixel decided using an electrical signal obtained based on such accumulated charge Sa.
However, according to the line showing the high luminance region 11, it is clear that the accumulated charge will have already reached the saturation level (the saturation point Px) before the time t3 is reached. Accordingly, for such high luminance region 11, it is only possible to obtain pixel values corresponding to an electrical signal with the saturation level from the long exposure image, resulting in the pixels being blown-out highlights.
For this reason, in the high luminance region 11, at a time before the time t3 is reached, for example, at the time t1 (a charge sweep start point P1) in FIG. 1, the charge accumulated in the light-receiving photodiode (PD) is swept out. This sweeping of charge does not sweep out all of the charge accumulated in the light-receiving photodiode (PD) and is assumed to reduce the charge to an intermediate voltage holding level that is controlled in the light-receiving photodiode (PD). After this charge sweeping process, a short exposure is taken again with the exposure length TS (t2 to t3). That is, a short exposure is taken for the period from a short exposure start point P2 to a short exposure end point P3 shown in FIG. 1. The accumulated charge (Sb) is obtained by this short exposure and the tone level of a pixel is decided based on the electrical signal obtained based on this accumulated charge (Sb).
Note that when a pixel value is decided based on the electrical signal based on the accumulated charge (Sa) obtained by the long exposure in the low luminance region 12 and the accumulated charge (Sb) obtained by the short exposure in the high luminance region 251, the pixel value level is decided based on the estimated accumulated charge for a case when exposure is carried out for the same time or on the calculation result when an electrical signal output value corresponding to such estimated accumulated charge is calculated.
In this way, by combining the short exposure image and the long exposure image, it is possible to obtain an image with high dynamic range and with no blown-out highlights.
However, with the configuration disclosed in Japanese Laid-Open Patent Publication No. 2000-50151, it is necessary to carry out processing that separately picks up the long exposure image and the short exposure image and combines the images.
In this way, although it is possible to generate a high dynamic range (HDR) image using a plurality of images in which the exposure length is changed, the processing based on such plurality of images has the following problem. Since the long exposure image has a longer exposure compared to the short exposure image, there is the problem of a drop in image quality due to occurrence of blurring caused by camera shake or subject movement and the occurrence of color loss and fake colors for a moving subject.